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ABSTRACT 

The purpose of this paper is to review some of the 
methods used for the impact evaluation of Demonstrator-Developer 
projects and to promote the meta-analysis approach. Four examples of 
attempts to collect and analyze impact data from adopters are 
presented, one of which used the meta-analysis approach. In this 
example, each adopter collected pre and post data using 
instrumentation specified by the parent project^ The- results were 
used to compute an effect size (a standardized growth measure) for 
each project using the performance of the control group in the 
original project. Using this approach, one not only is able to get an 
estimate, of the effectiveness of each separate adoption, but the data 
can easily be aggregated for ah estimate of the effectiveness of the 
parent project, (BW) 
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A META-ANALYSIS APPROACH TO IHPACT EVALUATION OF ADOPTIONS 



Introduction 

Demonstrator-Developer (DD) projects of the National Diffusion Network 
(NDN) have always been asked to supply evaluative data. The type of 
evaluative data usually falls into the following three catagbries — 
management , implementation , and impact • Evaluative data on management are 
quite straight forward and well documented. For the most part they include 
descriptions and counts of dissemination activities. Implementation data are 
estimates of whether adopters have implemented the key elements of the parent 
project. Impact evaluation data on the effectiveness of adoptions have been 
more elusive. 

The purpose of this presentation is to review some of the methods used 
for the impact evaluation of DD projects and to promote the meta-analysis 
approach. DD projects often have little or no control over the evaluation 
design (if any) that is used to evaluate the effectiveness of an adoption. 
Different adopters may use many varied evaluation designs. Further, most of 
these designs will not include a control group. For this reason, aggregating 
these data to obtain an overall picture of the effectiveness of adoptions is 
often impossible. 

Examples of Impact Evaluation of Adoptions 
The NDN heretofore has not enforced a requirement that DD projects gather 
the rigid type of impact data that JDRP would require. In most cases , impact 
data were riot collected at all or their collection was left up to the 
adopters . This section presents four examples of attempts to collect and 
analyze impact data from adopters . 
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In the first approach, data were not requested until after the project 
was implemented. Available data were collected and analyzed separately for 
each adoption. The remits were reported for each adoption. This approach 
was used to summarize the effectiveness of adoptions of the Talents Unlimited 
Program in 1979 (Chissom & McLean, 1980). 

The approach was effective in uncovering seven situations where impact 
data for the Talents Unlimited Program were available. The difficulty in this 
approach was that the evaluators had no control over the quality of the data 
collected. In fact , in most cases , the information on how the data were 
collected, how the students were selected, etc. was not available. 

In order to remedy these problems, a paper was written which specified 
the design requirements for establishing the validity of Talents Unlimited 
adoptions (McLean 5 Chissom, 1980a). The results of this approach were used 
for some of the Talents Unlimited adoptions during the 1979-80 school year 
(McLean & Chissom, 1980b) . During that year, 17 adopting school systems agreed 
to participate in the impact study using the guidelines as noted above (McLean 
& Chissom, 1980a) . Complete data were obtained from 10 of these adopters . 
Later, data from one o! the 10 were excluded because it was determined the 
data lacked validity cue to the testing conditions . 

This approach, although better than the first, still had a number of 
problems. The greatest of these was the requirement that every participating 
adopter use exactly the same design. In the end, about one-half of the 
original participants were excluded because this was impossible . 

A third approach involved a post hoc application of met a- analysis to 
adopter impact evaluation. This approach was used by Harpole (1982) to 
determine if the degree to which the key elements of the Pegasus-Pace reading 
program were implemented was related to the effect size (a standardized growth 
measure) of the participants. Each adopter provided pre- and post-test data 



which were used to determine effect size. Each adopter also completed a 
questionnaire which was used to determine the extent to which the key elements 
of the program were implemented ~. The main disadvantage of this approach was 
the post hoc nature of its application. It was not necessary to exercise 
control over the implementation of the evaluation at each site. 

The forth, and probably the best, example to be presented here is the one 
used by the State of Georgia/ s Educational Improvement Office and directed by 
Gearld Klein. The procedure was used to evaluate eight projects which have 
been state validated (five of which have also been validated by JDRP) and are 
receiving funds for state dissemination (Yeany & Okey , 1982). The design 
requires each adopter to agree to collect pre and post data using 
ins trumentation specified by the parent project. The results are used to 
compute an effect size for each project using the performance of the control 
group in the original project. 

During the 1981-82 year, the eight projects had 45 adoptions which 
involved over 135,000 students. The average effect size (the experimental 
groups T gain in standard deviation units * exceeding that of the original 
control group) was .7 3 and ranged from .30 to 1.60 (Yeany & Okey, 1982). ''An 
effect size of this magnitude corresponds to a shift of 26 percentile units 
from pre to post," (Yeany & ©key; 1982, p. 50). There are several obvious 
advantages to this approach. One not only is able to get an estimate of the 
effectiveness of each separate adoption, but the data can easily be aggregated 
for an estimate of the ef f ectivenss of the parent project . 

The Meta-analysis Approach 
As can be seen from the preceding examples , a meta-analysis approach 
holds great promise for the impact evaluation of DD projects . This sections 



provides a more in-depth look at this approach and suggests specific methods 
for implementing it* 

What Is Met a -analys ia ? 

Meta-analysis is a quantitative approach to analyzing the literature on a 
topic (Glass, McGaw, & Smith, 1981). it forces the researcher to 

systematically derive quantitative information from each research study being 
reviewed. Thus, a large number of studies on a particular topic can be 
quantified , analyzed, and findings summarised. 

Meta-analysis provides a vehicle for overcoming the problems of reviewing 
literature and as was seen in the previous section , it can also provide a 
means of evaluating the impact of a large number of adoptions. Meta-analysis 
provides the promise of assimilating a large number of studies into general 
conclusions . While there ate many methodological hurdles which must be 
overcome in its application , meta-analysis certainly has a number of 
advantages over previously utilized approaches. 

Whereas meta-analysis is a quantitative method, of analysis , its purpose 
is to provide general knowledge based on the individual results of several 
studies. For example, if project A was found to be successful in one study 
and not successful in another , meta-analysis has the potential to identify the 
reason for the different findings. This is done by considering possible 
concomitant variables in the meta-analysis . 

The quantification of a large number of studies presents a number of 
methodological problems . Among these are the identif i- cation of possible 
related variables , coding of concomitant vari- ables , measuring effect size , 
and analyzing Uhe results. All of these are problems of traditional research 
arid evaluation except measuring effect size. The effect size is a 
standardized difference score between the treatment and control groups . 
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Effect size is one of the concepts central to the application of meta-analysis 
methodology. 

One of the most perplexing problems in 'meta-analysis is the problem of 
the conclusion depending on the raechod of aggregation or analysis used. When 
the results depend on the method, one of the methods has led to an erroneous 
result. Glass et ai. (1981) report a study by Simpson on amphetamine therapy 
with hyperactive sixth-grade children which revealed opposite results 
depending on what method of data aggregation was used. Other examples are 
also available in the literature. 

Recovering Effect Size 

A problem unique to meta-analysis is that of determining effect size. It 
is interesting to note that effect size can be often be determined with many 
different design types. Obviously, fewer questions concerning the internal 
validity of the designs will arise when the design is specified as with the 
Yea / arid Okey example (1982). 

Effect size has been defined in different ways. In general, it can be 
defined as the posttest minus the pretest divided by the pretest standard 
deviation (Glass, et ai., 1981 & Yeany & Okey, 1982). This definition does 
not take into account the performance of the control group. Thus, it can be 
modified to be the experimental mean minus the control mean divided by the 
control standard deviation (Glass, et al., 1981 & Yeany & Okey, 1982). The 
important point is that it is defined the same way throughout a single study 
and the definition is made clear for interpretation purposes . 

As was noted earlier , effect sizes can be recovered for numerous 
situation, even where very different designs and analyses were used. Deaton 
(1982) indicated methods for recovering effect sizes for situations wbore 
differences were compared using t-tests, F-tests (ANOVA) , and nonparametric 



procedures ; He also provided techniques for recovering effect sizes for 
various correlational methods ; 

Analysis of Effect Sizes 

0nce effect sizes have been recovered, they can be used as the dependent 
measures in many types of analyses • The advantage of this is that such use 
can help provide information on when a program is effective and when it is 
not. For example , suppose that a reading program is effective for first 
graders but becomes less effective in the upper elementary grades. Using 
effect size as the dependent variable arid grade as the dependent variable in 
an ^analysis of variance will establish this. 

By converting the data from each site to effect sizes, the data can be 
aggregated arid almost any analysis performed. The problem of arriving at an 
overall conclusion about the effective of the program is simplified greatly. 
The effectiveness of individual sites can be compared and ranked (e.g., Yeany 
& Ohey, 1982). 

Summary 

Evaluating the impact of project adopters has long been a thorn in the 
side of DD projects. Meta-analysis does not eliminate ail of these problems 
but should certainly reduce them. As experience is gained with this approach 
to evaluating the impact of adoptions , the data can be aggregated riot drily 
over sites but over years • Analyses can be ddrie to determine the most 
effective implementat ion methods arid conditions and, at the same time, reduce 
the requirements for specific types of score scales. It could also be used to 
monitor adopters. 



- 6 - 



8 



References 

Chissom, B; S. & McLean, J; E; talents Unlimited programs: Technical report 
summarizing research findings * ERIC Document Reproduction Service No. 
ED 179 556, April, 1980. (also appears unabridged in the Congressional 
Record-Senate , September 11, 1980, pp. S12409-Si24il5 . 

Deaton, W. L. "Recovering effect sizes for meta-analysis. Paper presented at 
the Eleventh Annual Meeting of the Mid-South Educational Research 
Association, Hew Orleans, Louisiana, November, 1982. 

Harpole , J. Relationships of student reading achievement gains to key ele- 
ments of the PEGASUS^PACE reading program and to celected variables 
related to leadership, teachers, end time . Dissertation, The University 
of Alabama, University, Alabama , 1982. 

Glass, G. V. Primary, secondary, arid meta-analysis of research. Educational 
Researcher, 1975, 5* 3-5. 

Glass ^ G. V. ». HcGaw, B. s & Smithy M. L. Meta-analysis in social research. 
Beverly Hills - y CA: Sage Publications, 1981. 



McLean - % J. E. & Chissom ^ B. S . Ss * ablishing £he validity of a Talents Unlimit- 
ed program at the school le^el. ERIC Document Reproduction Service No. 
ED 181 065, 1980a. 

McLean , J. E. & Chissom, B. S. Talents Unlimited program, su mm ary -of- .research 
find;? ngs for 1979-80 . ERIC Document Reproduction Service No. ED 198 660, 
1980 b. 

Yeany, R. H. & Okey, 3. R. Evaluation report of the educational program dis- 
semination training ^ne£worti State of Georgia . Gerald Klein, Educational 
Improvements Office, Georgia State Department of Education. Atlanta, 
Georgia, December, 1982. 



